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InsideTrack® Coaching 


Earning a college degree is one of the primary pathways to economic success. Median weekly earnings of full-time workers 
with an associate degree in 2017 was 17 percent higher than full-time workers with a high school diploma only." Yet, large 
numbers of students who enroll in college do not complete a degree. Many programs and practices aim to improve college 
persistence and completion, including InsideTrack® Coaching. 


InsideTrack® Coaching provides proactive, personalized coaching to help students identify and overcome both academic and 
non-academic barriers to college persistence and graduation. This What Works Clearinghouse (WWC) report, part of the WWC's 
Supporting Postsecondary Success topic area, explores the effects of InsideTrack® Coaching on students’ persistence and degree 
attainment. The WWC identified 10 studies of InsideTrack® Coaching, one of which met WWC standards. The evidence presented 
in this report includes a study of the impact of InsideTrack® Coaching on four-year college students.’ 


What Happens When Students Participate in InsideTrack® Coaching?? 


The evidence indicates that implementing /nsideTrack® Coaching: 

* may increase students’ persistence in college 

* may result in little to no change in students’ degree completion 

Findings on InsideTrack® Coaching from the one study that meets WWC standards is shown in Table 1. For each outcome 
reviewed by the WWC, an effectiveness rating, the study findings, and the number of studies and students that contributed to 


the findings is presented. These findings are based on 3,527 students on the persistence outcome and 1,346 students on the 
attainment outcome. See Box 1 for a description of WWC effectiveness ratings. 


Table 1. Summary of findings on InsideTrack® Coaching from the study that meets WWC Standards 


Average performance Evidence meeting 
(study findings) WWC standards (version 3.0) 
Number of Number of 
Outcome Effectiveness rating Intervention group Comparison group studies students 
Credit accumulation and persistence Potentially positive effects 66.4% retained 61.4% retained ] 3,527 
Attainment No discernible effects 35.2% graduated 31.2% graduated ] 1,346 


Table Note: Average performance figures from study findings are based on one analysis conducted for each outcome, as reported by Bettinger & Baker (2014). These outcomes include retention in 
college at 12 months (credit accumulation and persistence) and completing a degree (attainment). The effects of Insidelrack® Coaching are not known for other outcomes within the Supporting 
Postsecondary Success topic area, including college access and enrollment; college attendance; academic achievement; and labor market outcomes. 
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BOX 1. HOW THE WWC REVIEWS AND DESCRIBES EVIDENCE 


The WWC evaluates evidence based on the quality and results of reviewed studies. The criteria that the WWC uses for evaluating 
evidence are defined in the Procedures and Standards Handbooks and the Review Protocols. The studies summarized in this report 
were reviewed under WWC Standards (version 3.0) and the Supporting Postsecondary Success topic area protocol (version 3.0). 


To determine the effectiveness rating, the WWC considers what methods each study used, the direction of the effects, and how many 
studies tested the intervention. The higher the effectiveness rating, the more certain the WWC is about the reported results and about 
what will happen if the same intervention is implemented again. The following key provides a link between effectiveness ratings and 
the statements used in this report: 


Effectiveness Rating Rating Interpretation Description of the Evidence 


Positive (or Negative) Effects The intervention is likely tochange Strong evidence of a positive effect, with no 
an outcome overriding contrary evidence 


Potentially Positive (or Negative) Effects — The intervention may change an Evidence of a positive effect with no overriding 
outcome contrary evidence 


No Discernible Effects The intervention may result in little | No affirmative evidence of effects 
to no change in an outcome 


Mixed Effects The intervention has inconsistent Evidence includes studies in at least two of 
effects on an outcome these categories: studies with positive effects, 
studies with negative effects, or more studies 
with indeterminate effects than with positive or 
negative effects 


How is InsideTrack® Coaching Implemented? 


The following section provides details of how InsideTrack® Coaching was implemented. This information can help educators 
identify the requirements for implementing InsideTrack® Coaching, and determine whether those implementation 
requirements would be feasible at their institutions. Information on InsideTrack® Coaching presented in this section comes 
from the study that meets WWC evidence standards (Bettinger & Baker, 2014), from the developer's website, and from 
correspondence with the developer. 


* Goal: Goals for InsideTrack® Coaching programs differ depending on the students served at each institution. These goals 
include increasing the college enrollment of admitted students and preparing students to succeed at a given institution; 
improving engagement, persistence, completion, and satisfaction of currently enrolled students; increasing rates of re-entry 
for students who have left a given institution; and supporting students’ and alumni’s career development. InsideTrack® also 
offers capacity building, training, and consulting services to institution staff; these services are outside the scope of this review. 


¢ Target Population: /nsideTrack® offers different coaching options for students enrolled in college, prospective students who 
have not yet enrolled, and recent graduates. All students are eligible to receive InsideTrack® Coaching, including traditional 
undergraduates, first-generation, low income, minority, graduate students, online and distance learners, adult learners, and 
military and military-connected students. 


* Method of Delivery: /nsideTrack® partners with universities to deliver its coaching program, supplying the personnel and 
technology. InsideTrack® provides coaching through phone, video, email, text, and mobile apps. The coaches gather relevant 
materials from the universities, including course syllabi, transcripts, and other student information to tailor the coaching to 
the student. InsideTrack® also supports institutions in building their own coaching programs. 


« Frequency and Duration of Service: The intensity of InsideTrack© Coaching 


depends on students’ needs and responsiveness to coaches’ outreach. Intensity of Comparison Group: In the one 
coaching may also depend on the institution's goals, such as whether /nsideTrack© study that contributes to this 
Coaching is targeted to help students have a strong start at college or to improve intervention report, students 
rates of completion. In the study that meets WWC standards, students generally in the comparison group had 
worked with coaches over two semesters and met with coaches at least five times. access to regular academic 


counseling and tutoring 
services that were already 
available at their college. 


* Intervention Components: The /nsideTrack© Coaching intervention has two 
primary components, as noted in Table 2. 
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Table 2. Components of InsideTrack® Coaching 


Key component —_InsideTrack® Coaching 


Coaching Insidelrack® Coaching offers multiple types of coaching including prospective student coaching, “strong start” coaching, retention coaching, 
and career coaching. Coaches provide tailored, technology-enabled support to students. This support includes helping students clarify 
their goals; identify academic and non-academic obstacles to success; keep track of institutional deadlines; find and access resources; build 
time management and study skills; and learn to strategize and advocate for themselves. Coaches focus on students’ lives outside of school, 
including personal time commitments, caregiving obligations, and finances, as well as their academic experience in school. 


uCoach® The uCoach® Technology Platform enables /nsidelrack® Coaching to deliver proactive, one-on-one coaching through multiple channels; 
Technology send automated messages at predefined intervals; track student engagement and coach observations; and predict when to reach out to 
Platform students on particular issues. Students can reach out to coaches through the platform at any time, and vice versa. Automated messages 


remind students about deadlines, opportunities, and resources, and the platform also includes self-directed resources that students can 
access for guidance on key topics. The platform can be integrated with other institutional systems (e.g., learning management systems), 
increasing possibilities for monitoring students’ needs and delivering tailored information and resources. 


What Does /nsideTrack® Coaching Cost? 


The cost of InsideTrack® Coaching varies depending on the objectives, scale, intensity, and duration of the program. The one 


s 


tudy summarized in this report (Bettinger & Baker, 2014) reported that in 2004 and 2007 InsideTrack® charged about $500 


per student per semester for its most comprehensive program. This included a fixed charge for the cost of customizing its 
program to a particular university and a variable charge that depended on the number of students coached. A more recent 
InsideTrack® Coaching program launched in 2013 was reported to cost $390 per student per semester.4 


The WWC also identified several cost components from the intervention description in the studies reviewed. This preliminary 


st of costs is not designed to be exhaustive; rather, it is designed to provide educators an overview of the major cost 


components of implementing /nsideTrack® Coaching. 


Personnel Costs: All /nsideTrack® coaches have college degrees, and most have advanced degrees. They undergo a rigorous, 
formal credentialing process that includes an average of over 100 hours of professional development per year. Coaches’ 
interactions with students are recorded, and they regularly receive feedback on these interactions. 


Facilities Costs: /nsideTrack® Coaching does not require physical facilities because most services are provided via phone, 
email, text messaging, and mobile apps. 


Equipment and Materials Costs: The uCoach® Technology Platform is one of the primary components of the /nsideTrack® 
Coaching intervention. This platform can be directly licensed to institutions, even if InsideTrack® Coaching services are not 
purchased. 


Costs Paid by Students or Parents: Students need access to personal technology (e.g., mobile phone, computer) to 
communicate with their coach. They do not pay user fees for coaching services. 


In-Kind Supports: Colleges provide information to InsideTrack® to help customize coaches’ interactions with students. This 
may include information such as student transcripts, course syllabi, or data on course performance. Colleges may also work 
to integrate their existing institutional systems (e.g., student information systems and learning management systems) with 

the uCoach® Technology Platform. 


Sources of Funding: Colleges typically contract with InsideTrack® to provide coaching services to their students. 
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For More Information: 


About InsideTrack® Coaching 
121 SW Salmon Street, Suite 800, Portland, OR 97204 
Web: https://www.insidetrack.com/. Phone: (800) 884-6371 


About the cost of the intervention 


Bettinger, E. P & Baker, R. B. (2014). The effects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation 
& Policy Analysis, 36(1), 3-19. Retrieved from https://eric.ed.gov/?id=EJ1019184 


RevUp Montana. (2017). RevUp Montana. Project executive summary: Final report to Montana Board of Regents. Retrieved from 
https://mus.edu/board/meetings/2017/Sept2017/TwoYear/BoR RevUp FinalReport_Sept2017.pdf 


Research Summary 


The WWC identified 10 studies that investigated the effectiveness of InsideTrack® Coaching (Figure 1): 

+ 1 study meets WWC group design standards without reservations 

+ Ostudies meet WWC group design standards with reservations 

+ 8 studies do not meet WWC group design standards 

+ 1 study is ineligible for review 

The WWC reviews findings on an intervention’s effects on eligible outcome domains from studies that meet WWC group 
design standards, either with or without reservations. Based on this review, the WWC generates an effectiveness rating, which 


summarizes how the intervention impacts, or changes, a particular outcome domain. Findings from studies that either do not 
meet WWC standards or are ineligible for review do not contribute to the effectiveness ratings. 


The one study of InsideTrack® Coaching that meets WWC group design standards reported findings on (a) credit accumulation 
and persistence and (b) attainment. This study did not report findings on the following four domains in the Supporting 
Postsecondary Success topic area: access and enrollment, college attendance, academic achievement, and labor market 
outcomes. Citations for the 10 studies reviewed for this report are listed in the References section, which begins on page 8. 


Figure 1. Effectiveness ratings for InsideTrack® Coaching 


study meets WWC studies meet WWC studies do not 1 study is 

standards without standards with meet WWC ineligible for 

reservations reservations standards review 
Contributes to Effectiveness Ratings Do Not Contribute to Effectiveness Ratings 


SSS SSS og 


InsideTrack® Coaching has potentially positive effects on credit accumulation and persistence 


The WWC determined that one study that meets WWC group design standards without reservations showed evidence of a positive and 
statistically significant effect of InsideTrack® Coaching on college persistence (Bettinger & Baker, 2014). 


InsideTrack® Coaching has no discernible effects on attainment 


The WWC determined that one study that meets WWC group design standards without reservations showed evidence of an 
indeterminate effect of InsideTrack® Coaching on college degree completion (Bettinger & Baker, 2014). 
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Main Findings 


Table 3 shows the findings from one InsideTrack® Coaching study that meets WWC evidence standards, and includes WWC 
calculations of the mean difference, effect size, and improvement index. Note that because only one study of InsideTrack® 
Coaching meets WWC standards, the summary information for the outcome will match the individual study findings for that 
outcome. Based on findings from that study, the effectiveness rating for the credit accumulation and persistence outcome is 
potentially positive effects, indicating that there is evidence of InsideTrack® Coaching’ positive effect with no overriding contrary 
evidence. This finding is based on 3,527 students. The effectiveness rating for attainment outcomes is no discernible effects. 
This finding is based on 1,346 students. 


Table 3. Findings from the study of InsideTrack® Coaching by outcome domain 


Mean 
(standard deviation) WWC calculations 
Intervention Number of Mean Effect += Improvement 

Measure (study) Study sample Sample size group Students difference size Index p-value 
College retention (9) 7 lotteries / 12 3,527 66.4 61.4 5.0 0.13 +5 <.01 
(Bettinger & Baker, month follow-up students 
2014)? 

Outcome average for credit accumulation and persistence 0.13 +5 
Completed a degree 3 lotteries / 24 1,346 35.2 31.2 4.0 0.11 +4 <.10 
(%) (Bettinger & Baker, month follow-up students 
2014) 

Outcome average for attainment 0.11 +4 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors the comparison 
group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are given the intervention (measured 
in standard deviations of the outcome measure). The WWC-computed average effect size is a simple average rounded to two decimal places; the average improvement index is calculated from the 
average effect size. The improvement index can be interpreted as the expected change in percentile rank for an average comparison group student if that student had received the intervention. The 
statistical significance of the domain average was determined by the WWC. Some statistics may not sum as expected due to rounding. 


*For Bettinger & Baker (2014), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The intervention group mean was calculated by 
adding the OLS coefficient to the unadjusted comparison group mean. This study is characterized as having a statistically significant positive effect on credit accumulation and persistence because 
the estimated effect is positive and statistically significant. The study is characterized as having no discernible effect on attainment because the mean effect reported is not statistically significant. For 
more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), page 26. 


In What Context Was InsideTrack® Coaching Studied? 


The following section provides information on the setting and participants involved in the one study of InsideTrack® Coaching 
that meets WWC evidence standards. This information can help educators understand the context in which the study of 
InsideTrack® Coaching was conducted so that they can better determine whether the program might be suitable for their 
setting. 
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WHERE THE STUDY WAS CONDUCTED 


Til 1 study, 3 27 students in 3 four-year colleges 


GENDER 


Female 


GRADES 0 nu 2 & 
Postsecondary (PS) 


Details of Each Study that Met WWC Standards 


This section presents details for the study of InsideTrack® Coaching that meets WWC standards. These details include the full 
study reference, findings description, findings summary, and description of study characteristics. A summary of findings for 
each outcome domain examined is presented in Table 4. Table 5 provides a description of the study characteristics; it includes 
contextual information around the study setting, methods, sample, intervention group, comparison group, outcomes, and 
implementation details. For additional information, the reader should refer to the original study. 


Research details for Bettinger & Baker (2014) 


Bettinger, E. P & Baker, R. B. (2014). The effects of student coaching: An evaluation of a randomized experiment in student 
advising. Educational Evaluation & Policy Analysis, 36(1), 3-19. Retrieved from https://eric.ed.gov/?id=EJ1019184 


Additional Source: 


Bettinger, E. P, & Baker, R. B. (2011). The effects of student coaching in college: An evaluation of a randomized experiment 
in student mentoring (NBER Working Paper No. 16881). Cambridge, MA: National Bureau of Economic Research. 
Retrieved from http://www.nber.org/papers/w16881 


Findings from Bettinger & Baker (2014) show evidence of a potentially positive effect of InsideTrack® Coaching on credit 
accumulation and persistence (Table 4). The improvement index is a measure of the effect of the intervention. For example, 
the improvement index of +5 means that the percentile rank of the average comparison group student would improve by 5 
points on credit accumulation and persistence outcomes if they received InsideTrack® Coaching. This finding is based on one 
outcome and 3,527 students. The finding on attainment shows evidence of no discernible effects. This finding is based on 
1,346 students. 


Table 4. Summary of findings from Bettinger & Baker (2014) Meets WWC group design standards without reservations 
Study findings 
Average Improvement Statistically 
Outcome domain Sample size effect size index significant 
Credit accumulation and persistence 3,527 students / 7 lotteries 0.13 +5 Yes 
Attainment 1,346 students / 3 lotteries 0.11 +4 No 
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Table 5. Description of study characteristics for Bettinger & Baker (2014) 


WWC evidence Meets WWC Group Design Standards Without Reservations. [he study is a randomized controlled trial (RCT) with low 

rating attrition in seven of the 17 lotteries described. In the other 10 lotteries, students were reassigned to treatment and comparison groups 
after randomization took place, which compromised random assignment. Because baseline equivalence was not established on the 
analytic sample, findings from these 10 lotteries received a rating of Does Not Meet WWC Group Design Standards. 


Setting The study was conducted with 17 different cohorts in eight participating universities during the 2003-2004 and 2007-2008 school years. 


Methods Each institution had its own eligibility criteria and provided a list of potential students for Insidelrack® to randomly assign into two 
groups. Insidelrack® then performed two types of randomization: 


(1) For institutions that wanted equally sized groups (7 of 17 lotteries, referred to by the authors as”50/50 lotteries”), Insidelrack® 
created two randomly assigned groups of approximately equal size and the institution decided which of the two groups would receive 
the intervention through a coin flip. Following the coin flip, the institution was notified which students were in each group. Insidelrack® 
monitored the randomization to make sure that the two groups were balanced across observable characteristics. In some cases, students 
were moved between groups to achieve balance; however, this rebalancing occurred prior to the coin flip. 


(2) For institutions that wanted a smaller control group (10 of 17 lotteries), the institution provided Insidelrack® with a predetermined 
size for the control group, and /nsidelrack® then randomly assigned two groups to meet those size restrictions. There was some 
rebalancing in these lotteries as well, but no coin flip followed the rebalancing (because the largest group received treatment). 


Across the 17 lotteries, 8,049 students were assigned to Insidefrack® Coaching services and 5,506 students were assigned to the 
comparison group. 


Study sample Seventeen lotteries were conducted across eight postsecondary institutions in the 2003-2004 and 2007-2008 school years. Only seven 
of the 50/50 lotteries were determined by the WWC to be effectively randomized and produce findings that meet standards. The study 
sample for the retention outcome includes seven 50/50 lotteries. At baseline, this sample included 1,768 students in the Insidelrack® 
Coaching group and 1,768 students in the comparison group. The study sample for the attainment outcome included three 50/50 
lotteries. At baseline, this sample included 675 students in the Insidefrack® Coaching group and 675 students in the comparison group. 


he analytic sample for the retention outcome included seven 50/50 lotteries comprised of 1,763 students in the InsideTrack® Coaching 
roup and 1,764 students in the comparison group. The sample for the degree completion outcome included three lotteries comprised of 
73 students in the Insidelrack® Coaching group and 673 students in the comparison group. 


= 


g 

6 

Characteristics of the study sample were not presented separately by lottery. The analytic sample included 47 percent female and 53 
percent male students. 

S 


Intervention group Students in the intervention condition were paired with an Insidelrack® coach, who worked to help students prioritize their studies, plan 
for academic success, and identify and overcome barriers to academic success. Significant time was spent assessing students lives outside 
of school in such areas as personal time commitments, primary caregiving responsibilities, and financial obligations. In addition to regular 
contacts, coaches sometimes had access to course information and student performance in their specific courses. This information was 
used in an algorithm which directs coaches to specific issues that need to be addressed. Coaches generally worked with students over two 


semesters. 


Each coach communicated with his or her students via phone, email, text messages, or social networking sites. Students’ engagement 
with Insidefrack® coaches was not mandatory. About 98 percent of the students in the /nsidelrack° Coaching group received at least 
one brief contact from a coach that typically lasted less than five minutes. About 77 percent of the students in the /nsidelrack® Coaching 
group received at least five contacts of less than five minutes each. Insidelrack® coaches also held longer meetings with students to 
address topics and identify next steps. 


Comparison group —_ The comparison condition received no individualized coaching through Insidefrack®. All students had access to regular support services 
provided through the college. 


Outcomes and The study measured two primary outcomes: college retention and degree completion. College retention was measured using semester 

measurement enrollment data provided by participating institutions. Retention outcomes were measured at 6, 12, 18, and 24 months after initial 
enrollment. In this report, we report the 12 month retention outcome, as this is the most distal outcome reported for the seven 50/50 
lotteries. Degree completion was defined as the completion of a certificate, an associate’s degree, or a bachelor's degree. Degree 
completion was measured 24 months after initial enrollment, and data were available for three of the 2003-2004 lottery cohorts, all of 
which were at four-year colleges. 


Additional Insidefrack® provided coaches with a large library of tools and resources. Coaches were trained in using these proprietary methodologies 
implementation and programs to help students navigate decisions. Coaches received feedback from InsideTrack® staff on the content and tone of their 
details calls, and ongoing professional development was available. 
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Endnotes 


' Bureau of Labor Statistics. (2018). Measuring the value of education. Washington, DC: Author. Retrieved from https://www.bls.gov/careerout- 
look/2018/data-on-display/education-pays.htm 


* The descriptive information for this intervention comes from Bettinger & Baker (2014). The What Works Clearinghouse (WWC) requests developers 
review the intervention description sections for accuracy from their perspective. The WWC provided the developer with the intervention description 
in December 2018 and the WWC incorporated feedback from the developer. Further verification of the accuracy of the descriptive information for this 
intervention is beyond the scope of this review. 


>The literature search reflects documents publicly available by April 2019. Reviews of the studies in this report used the standards from the WWC 
Procedures and Standards Handbook (version 3.0) and the Supporting Postsecondary Success review protocol (version 3.0). The evidence presented 
in this report is based on available research. Findings and conclusions could change as new research becomes available. 


“The RevUp Montana study (2017), which received a rating of Does Not Meet WWC Group Design Standards, reported the annual cost of InsideTrack® 
Coaching to be $781 per student. 


° Data required to establish baseline equivalence were not available for these 10 lotteries; although baseline data were available for a subset of lotteries, 
it was unclear which lotteries had all required data points to establish baseline equivalence. 
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